Skip to content

fix(test): create-batch --run without --wait exits 0 on a fully successful dispatch#267

Merged
jangjos-128 merged 1 commit into
TestSprite:mainfrom
kshitij-heizen:fix/create-batch-run-nowait-exit
Jul 22, 2026
Merged

fix(test): create-batch --run without --wait exits 0 on a fully successful dispatch#267
jangjos-128 merged 1 commit into
TestSprite:mainfrom
kshitij-heizen:fix/create-batch-run-nowait-exit

Conversation

@kshitij-heizen

@kshitij-heizen kshitij-heizen commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes test create-batch --run without --wait always exiting 1 even when every trigger succeeds — which broke the documented exit-code contract and failed CI pipelines on fully successful dispatches.

The no-wait path returns each result with the trigger response's status, which is queued by design (the field is documented as "Terminal status if --wait; queued if no --wait"). But the exit-code logic only recognized terminal statuses:

const allPassed = batchRunResults.every(r => r.status === 'passed'); // queued → false
if (allPassed) return;
...
throw new CLIError(`Batch run finished: ${...} of ${...} run(s) did not pass.`, 1);

Every no-wait result is queued, so allPassed was always false and the command fell through to exit 1 with a misleading "N of N run(s) did not pass" — even on a fully successful dispatch. It was also internally inconsistent: single test run <id> without --wait exits 0 on a successful queued dispatch.

Fix

In the no-wait case, "success" means every trigger was dispatched without error (statuses are non-terminal by definition). The exit-code block now branches on opts.wait:

const failing = opts.wait
  ? batchRunResults.filter(r => r.status !== 'passed')   // --wait: terminal 'passed'
  : batchRunResults.filter(r => r.error !== undefined);  // no --wait: dispatched OK
if (failing.length === 0) return; // exit 0

Trigger errors in no-wait mode keep today's aggregation (uniform specific code when shared, else exit 1), and the "did not pass" message becomes "N of N trigger(s) failed" when --wait is absent. The stderr text summary is likewise corrected for no-wait mode — it now reports N/N triggered instead of misreading a successful dispatch as 0/N passed. --wait semantics are unchanged.

Reproduction (before this PR)

testsprite test create-batch --plans ./plans.jsonl --run --output json
# stdout: results[] with every entry status:"queued", no errors
# stderr: "Batch run finished: 2 of 2 run(s) did not pass."
# exit code: 1  ← CI marks a fully successful dispatch as FAILED

Related issue

Fixes #161

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Documentation only
  • Build / CI / chore

Checklist

  • PR targets the main branch.
  • Commits follow Conventional Commits.
  • npm run lint and npm run format:check pass.
  • npm run typecheck passes.
  • npm test passes and coverage stays above the 80% gate (1618 tests; 88.34% stmts / 85.42% branch).
  • New behavior is covered by unit tests (mock-based; no network or credentials required).
  • No secrets, API keys, internal endpoints, or personal data are included.
  • User-facing changes are reflected in DOCUMENTATION.md (create-batch now documents the no-wait exit-0 contract, mirroring test run).

Notes for reviewers

  • Three new specs in test.create-batch-run.spec.ts:
    • all-queued (--output json) → resolves without error and never polls (pollCount === 0);
    • all-queued (--output text) → summary reads 3/3 triggered, with no passed / did not pass wording;
    • partial trigger failure (2 queued + 1 NOT_FOUND) → still exits non-zero, full results envelope intact.
  • The two success specs fail against the pre-fix exit-code block and pass with it (verified by reverting the source hunk); the partial-failure spec passes in both, guarding the error path. Existing no-wait specs only exercised error scenarios and never asserted the success exit code — which is how this slipped through.
  • The change is scoped to the exit-code + text-summary block in runBatchRun; JSON output shape, --wait semantics, and all other exit codes (6/7/11) are unchanged.

Summary by CodeRabbit

  • Bug Fixes

    • Improved test create-batch --run behavior when used without --wait.
    • Queued runs now report successful dispatches instead of being incorrectly treated as passed or failed.
    • Immediate trigger errors still produce a non-zero exit code while preserving successful and failed results in the output.
    • Text summaries now accurately show trigger counts and errors.
  • Documentation

    • Clarified batching, waiting behavior, polling, and exit-code semantics.

…ssful dispatch

The no-wait fan-out returns each result with the trigger response's status,
which is `queued` by design (non-terminal — the field is documented as
"Terminal status if --wait; queued if no --wait"). But the exit-code logic
only recognized terminal statuses:

    const allPassed = batchRunResults.every(r => r.status === 'passed');

Every no-wait result is `queued`, so `allPassed` was always false and the
command fell through to exit 1 with a misleading "N of N run(s) did not
pass" — even when every trigger was dispatched successfully. This broke the
documented exit-code contract (the DOCUMENTATION.md create-batch example
uses `--run --max-concurrency 4 --output json`, no `--wait`), failed CI on
fully successful dispatches, and was internally inconsistent with single
`test run <id>` without `--wait`, which exits 0 on a successful `queued`
dispatch.

In the no-wait case "success" means every trigger dispatched without error
(statuses are non-terminal by definition). The exit-code block now branches
on `opts.wait`:

    const failing = opts.wait
      ? batchRunResults.filter(r => r.status !== 'passed')
      : batchRunResults.filter(r => r.error !== undefined);
    if (failing.length === 0) return; // exit 0

Trigger errors in no-wait mode keep today's aggregation (uniform specific
code when shared, else exit 1), and the "did not pass" message becomes
"N of N trigger(s) failed" when `--wait` is absent. The stderr text summary
is likewise corrected for no-wait mode — it now reports "N/N triggered"
instead of misreading a successful dispatch as "0/N passed".

Adds three specs in test.create-batch-run.spec.ts: all-queued (json) →
resolves without error and never polls; all-queued (text) → "N/N triggered"
summary, no "passed"/"did not pass" wording; partial trigger failure → still
exits non-zero with the full results envelope. The two success specs fail
against the pre-fix exit-code block and pass with it. Existing no-wait specs
only exercised error scenarios and never asserted the success exit code,
which is how this slipped through.

Fixes TestSprite#161
@kshitij-heizen
kshitij-heizen force-pushed the fix/create-batch-run-nowait-exit branch from fe26b4a to 8f463b3 Compare July 22, 2026 07:11
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Changes

Batch run no-wait behavior

Layer / File(s) Summary
No-wait dispatch reporting and exit classification
src/commands/test.ts
No-wait runs now summarize triggered and errored requests, treat queued results without errors as successful, and use trigger-specific failure messages and exit codes.
No-wait behavior tests and documentation
src/commands/test.create-batch-run.spec.ts, DOCUMENTATION.md
Tests cover queued success, text output, and immediate trigger failures; documentation describes --wait and no-wait semantics.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: ruili-testsprite

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly states the primary behavioral fix for no-wait create-batch runs.
Linked Issues check ✅ Passed The changes address #161 by making no-wait dispatches exit 0 on queued triggers, preserving wait behavior, and adding the requested tests and docs.
Out of Scope Changes check ✅ Passed All changes are directly tied to the no-wait exit-code fix, its messaging, tests, and documentation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint install timed out. The project may have too many dependencies for the sandbox.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/commands/test.ts`:
- Around line 2938-2942: Update the no-wait error branches around the allTimeout
and corresponding failure handling to use trigger/dispatch terminology instead
of batch run terminology, including the messages and any related counts or
descriptions. Preserve the existing behavior and leave the default wait-path
wording unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d507dbbb-ff90-47dd-977e-f2e909389ebb

📥 Commits

Reviewing files that changed from the base of the PR and between 2708a40 and 8f463b3.

📒 Files selected for processing (3)
  • DOCUMENTATION.md
  • src/commands/test.create-batch-run.spec.ts
  • src/commands/test.ts

Comment thread src/commands/test.ts
Comment on lines 2938 to 2942
if (allTimeout) {
throw new CLIError(
`All ${batchRunResults.length} batch run(s) timed out after ${timeoutSeconds}s.`,
7,
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep no-wait failures in trigger terminology.

These branches can run with wait: false, but report timed-out/failed runs. No-wait only dispatches triggers; the default branch at Line 2963 already uses the correct terminology.

Proposed fix
- `All ${batchRunResults.length} batch run(s) timed out after ${timeoutSeconds}s.`
+ opts.wait
+   ? `All ${batchRunResults.length} batch run(s) timed out after ${timeoutSeconds}s.`
+   : `All ${batchRunResults.length} trigger(s) timed out after ${timeoutSeconds}s.`

- `Batch run finished: ${failing.length} run(s) failed with exit code ${uniformCode}.`
+ opts.wait
+   ? `Batch run finished: ${failing.length} run(s) failed with exit code ${uniformCode}.`
+   : `Batch run trigger finished: ${failing.length} trigger(s) failed with exit code ${uniformCode}.`

As per path instructions, text should match the underlying dispatch/trigger outcomes.

Also applies to: 2953-2957

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/commands/test.ts` around lines 2938 - 2942, Update the no-wait error
branches around the allTimeout and corresponding failure handling to use
trigger/dispatch terminology instead of batch run terminology, including the
messages and any related counts or descriptions. Preserve the existing behavior
and leave the default wait-path wording unchanged.

Source: Path instructions

@jangjos-128

Copy link
Copy Markdown
Contributor

Right shape for #161: the exit-code tail now branches on --wait — no-wait success = every trigger dispatched (a queued result counts) → exit 0, the --wait path stays behavior-preserving, and the documented batch semantics hold (all-conflict → 6, deferred/timeout → 7, empty-batch safe); partial no-wait failures now surface their specific error code rather than a generic 1. 3 tests covering each behavior. Thanks @kshitij-heizen!

@jangjos-128
jangjos-128 merged commit 251b593 into TestSprite:main Jul 22, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Hackathon] create-batch --run without --wait always exits 1 even when every trigger succeeds

3 participants